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ABSTRACT 

Problems associated with the analysis of data 
collected using the Solomon Four Group Design are discussed. The 
design includes an experimental group and a control group that have 
been pretested and posttested, and an experimental and a control 
group that have been posttested only. A sample problem is approached 
in three different ways. First, thedata are divided into two sets 
(those both pretested and posttested, and those posttested only) and 
analyzed using a' simple t, test. Second, the problem is visualized as 
a repeated measures design wherein the pretest is the first measure 
and the posttest is the second measure. Using this approach, the data 
can be analyzed with regression techniques. Third, if the design were 
seen as a six group design, a one-way lay-out or a three-way lay-out 
are possible. Both options are discussed, includirg the appropriate 
restrictions and multiple comparison techniques. (CM) 
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a Solomon Four Group Design 

John 0. Williams and Isadore Newman 

Solomon (1949) devised a design to control threats to design validity 
(Campbell and Stanley, 1966). Using the notation'^ Campbell and Stanley, 
the four groups can be diagrammed as 
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<Mhe four groups are respectively Group One: An experimental groups that has . 
been pretested and posttested; Group Two: A control group that has been pre- 
tested and posttested; Group Three: An experimental group that has been post- 
tested only; and Group Four: A control group that has been posttested only. ^ 

Campbell and Stanley state, "There is no singular statistical procedure 
which makes use of all six sets of observations simultaneous, ly." (p. 24) 

The Solomon Four Group Design, while very simple conceptually, can be 
very misleading depending upon the statistical analysis. Campbell and 
Stanley (1966) have a preferred approach, in which they set up a 2x2 fac- 
torial design. 
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In this design only the posttest scores are statistically analyzed. This 



procedure does not allow one to control for the pretest scores in groups 
1 and 2, thereby losing some power; it does however estimate the effects 
of treatments that are independent of individuals having a pretest and 
treatment pretest? interaction. It also tests for the effects, of pretesting, 
independent of treatment and pretest-treatment interaction, on pcsttest scores. 
Finally, •the approach estimates the effects of ^pretest-treatment interaction, 
on posttest scores. £ 

One of the advantages o c writing specific regression motels which reflect 
research questions is that one is less likely to have a statistical answer 
that is unrelated to the researcher's question of interest. The following are 

X 0 

a variety of regression models which will reflect potential research questions 
that can be ascertained from the Solomon Four Group Design. It should be 
remembered that there is not one correct answer. 

Recently, Newman, Benz, and Williams (1980) devised a way to analyze data 
that, b/ extension; right be applied to Soiomon type designs. A unique property 
of tms tc:nnique is that, the statement by Campbell and Stanley not with- 

Lancing, a sincle statistical procedure can be employed which makes use of 
all si* sets of observations simultaneously. On the other hanc, the solution(s) 
may prove to be no mere satisfactory than existing possibilities that split 

c 

tne aatd into two sets. In the erjd, the Solomon Four Group design may prove 

tc re one of those recalcitrant research situations that leave the would be 

s 

«. 

a^l/sts foundered on the shoal of a simple design whose simplicity is only 

h deception. 

<>n$ider Vre following research situation. Five people in each group 
•av^ scores such that one experimental group has been pretested and post- 
tested and one experimental group has been pos£tested only. Two similarly 

♦estPd con'rol groups are also included. Data for such'a situation are 

ft 

3 / 



given in Table 1. 



Table 1 

Data for a Solomon Four Group Design 



Experimental : 
Group One 
Pretest Posttest 



7 
5 

12 
6 



15 
12 
10 
17 
11 



Control : 
Group Two 
Pretest Posttest 



5 
4 
4 
6 
6 



8 
7 
8 
6 
6 



Experimental : 
Group Three 
Posttest 

13 
10 
12 
11 
14 



Con t ml : 
Group Four 
Posttest 

9 
8 
6 
3 
4 



Several different approaches might be trijed. One approach would be to 
divide tne data into two sets: Groups One and Two (those who were both pretested 
and posttested) as one set, and the posttested only groups (Groups Three and 
Four) as the second set. The latter. set can be simply tested by the use of the 
t test: 

t - 4/24 (p'.Q5). 

The former data set (Groups One and Two) 'can be conceived either as a 
ropeated measures design cr as a problem that can be approached through the 
analysis of covariance (or related techniques such as residual gain analysis). 

'To approach the problem first as an analysis of covariance, the following 
variables can be defined: 

/ = the criterion, or posttest score; 

* 

- the pretest score;' 
X 2 - 1 if tne score' is from the experimental group, 0 if the score is from 
the control group; 

x 3 = 1 if the score is from the control group, 0 if the score is from 
the control group. 
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Then either of two full models can be used: ^ 

Y = b 1 X 1 +b 2 X 2 +b 3 X 3 +e 1 , (1) 

or 

Y = b Q +b 1 X 1 +b 2 X 2 +e 1 . (2) 

Equation 2 utilizes the unit vector in the process of generating a constant 

2— 

whereas equation 1 does not. Either model will yield the same R value. 

The restricted model -(with equation 2 as the full model) is of the form: 

(3) 



For this (data set ^ 



.79379, R3 = .42334 



, F = (.79379 - .42334)/! ,= 12.58, p<„.05. 
(1 - .79379)/7 



Using a Repea te d Measures Approacn 
If the problem is visualized as a repeated measures design 'wherein the pretest 

c 

is the first measure and the posttest-is the second measure, then the design is 

like the Type I design shown in Lindquist (1953) and can be achieved through a 

regression approach (WHliams, 1974). For a regression formulation, see Table 2. 

Table 2 _ * 

De'^iqn Matrix for a Repeated Measures Problem 
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Here, Y = the criterion test score; 

P 1 thru P 1Q are binary coded person vectors (1 if the person, 0 otherwise); 
X.^ = 1 if the score comes from a person in the experimental group, 0 
otherwise; 

X 2 = 1 if score comes frpm a> person in the control group, 0 otherwise; 

X- = 1 if the score occurs with a pretest situation, 0 otherwise; 

3 ^ 

X. - 1 if the score occurs with a posttest situation, 0 otherwise; and 

x 5 = W ^ 

Several models can be used to generate an analysis. The use-of the following 
is instructive:. 



Y = b 0 +b 1 P 1 +b ? P 2 +...+b g P g +e 3 ; (4) 

(or alternatively, Y - bjPj+^V' ' ' +b 10 P 10+ e 3) 

Y = b Q +b 1 X 1 +e 4 5 " (5) 

(6) 



V b l X l +b 3 X 3 +e 6 ; {7) 



i 0 



' Y - b 0+ b 3 X 3+ e 5 ; 

Y = b 0+ b lXl+ b 3 X 
, V = b 0+ b 1 X 1+ b3X 3+ b 5 X 5+ e 7 ; (8) 

and 

v - V b r] fb 2 P ? f --- +b 9 P 9 +b 10 X 3 +b llV e 8- (9) 
>r fie preceding, R 2 - .54297; 

Rg = .31250; 

R? .31250; 

0 

= .62500; 

2 2 
Rg = .70312 and P g = .93359. 



What might have occured if a-model of the followinq form were used? 

Y = V^ P l +b 2 P 2 + --- +b 9 P 9 +b 10 X l^ b li X 3 +b 12V e 9- 
It wo^id not sensibly yield R 2 = .54297 + .70312 = 1.24609. Such a model would 

fail because the effect for experimental -control is "nested" in the subject 

(cr person) effect. To test for the experimental -control effect, 
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F*= = .31250/1 = 10.85, 

-1—-^ (. 54297-. 31250)/(10-2) * 

~ (Rj-RgJ/tP-l-D 
p<.05. 

To test for the test-retest effect, 

F = R 6^ 

(1-R*)7(N-P-1-1) 

= .3 1250 - 37.6b, p< . 01 . 

.06641/8 - 7 

The interaction is tested by 

« •» 

F. = (R?-R?)/1 - 88 (.70312-. 62500)/! = 9.41, p<.05. 

- 8 -^ .06641/8 

(1-Rq)/(N-P-1-1) 

Note that the interaction effect can be conceptualized as actually being 
additional evidence for the experimental effect. The higher increases in 
tne experimental group will show up in part as interaction for a repeated 
measures desiqn. 

The usual summary table for the repeated measures design can be constructed, 

The summary table is shown in Table 3. 

Table 3 

Sumroar/ Table for Repeated Measures Desiqn 



Subjects 

Experimental -Control 
error (a) 

Within Subjects 
test-retest 
interaction 
error (b) 



c 



Total 



df 




SS 




MS 


F 


9 




139.00 








1 




80.00 




80.00 


10.85 


8 




59.00 




- 7.375 




10 




117.00 








1 




80.00 




80.00 


37.65 


1 




20.00 




20.00 


" 9.41 


8 




17.00 




2.125 




19 




256.00 









•i? ing All Six Groups Simultaneously 
As the Solomon design is approached, several conceptual issues ensue. Is 
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'this to be seen as a six group design with attendant solutions? If the 
researcher opts for a six group design, person vector information needs to 
be excluded. Indeed, this was also true in the previous section. At no 
time were the four groups 'and person vectors used simultaneously; if it 
were, the R 2 was theoretically to be 1.24609, obviously an impossibility. 
If a six group design is to be used, what dimensions would be appropriate? 
This could be considered to be a one-way lay-out, a two-way lay-out, or a 
three-way lay-out (but with two missing cells) only the one-way and three-way 
layouts are discussed here. First hypotheses with a one-way lay-out as addressed 

Consider the following variables: 
v = the criterion score; 

X = 1 if the score is a pretest score from a member of the experimental 

group, 0 otherwise; 
X 2 = 1 if the score is a posttest score from a member of the experimental 

group that has been pretested, 0 otherwise; 
X 3 = 1 if the score is a pretest score from a member of the control group, 

0 otherwise; 

/ i if the score -is a nosttest score from a member of the control group 

4 

that was pretested, 0 otherwise; 

X = '1 if the score is from a member of the experimental group that was not 

5 

pretested, 0 otherwise; and 

X c - 1 if the score is from a member of the control group that was not pre- 

6 

tested, 0 otherwise. ^ 

for the six group situation, the full model is: 

» 

V = b 1 X 1+ b 2 X 2+ b 3 X 3 tb 4 X 4+ b 5 X 5+ b 6 X 6+ e 9 . (10) } 
At least two different sets of restrictions might make sense in addressing 
the Solomon design. One such set would be b^-b^b^-b^ which addresses the 
hypothesis Y 2~^ Y 4~ Y 3' as thP h yP° thesis that the 9 ains in the twice tested 

8 ' 



"experimental and control groups are equal; also, tfie second restriction is 

ja c =b c as the once tested experimental groups have equal ireans: Y>Y fi . 
■56 v 

The first restriction can be rewritten as b^b^-bj+b^ Placing these 
two restrictions on the Full Model: 

Y = b 1 X 1+ (b 4 -b 3+ b 1 ^ 2+ b 3 X 3+ b 4 X 4+ b 5 X 5+ b 5 X 6+ e 10 (11) 

Y = b 1 (X 1 +X 2 )+b 3 (X 3 -X 2 )+b 4 (X 4 +X 2 )+b 5 (X 5 +X 6 )+e 10 . (12) , 
Letting D 1 = X^X^; 

°3 W aPd 

D„ * X +X r , the restricted model is: 
4 5 o „ 

Y = b 1 D 1+ b 3 D 2+ b 4 D 3+ b 5 D 4+ e. 10 . (13) 

Here, R^ Q = .71183; =. .42882. 
2 2 

F = (R 1Q -R 13 )/2 = ,28301/2 = 11.79, p<.01. 

(i^ 0 )/(m.6):^ 1 - 71183,/z4 ■ 

This F test tests simultaneously Y\ = V T 3 and V 7 6; placing both sets 

of restriction s-a llows the rejection o f thp null hypotheses . — If these restric- 

tic : :s are equivalent to hypotheses the researcher had in mind, then there is no 
f urther problem. Translating the meaning of these two hypotheses into English 
may leave the researcher somewhat uneasy; however, one attempt at a translation 
into English is: ft is not simultaneously true that there is no" di fferences 
in the means of the non-pretested group and that there is no differences in 

the gains of the pre-tested groups. > * 

f **■ i 

One approach would be 'to tezt each of these hypotheses separately and using 
Dunn's ( 196 1) test for multiple comparisons. Imposing the first restriction separately, 
:b 2 -b L =b 4 -b 3 ) yields Y = b 1 X 1 +(b 4 -b3+b 1 )X 2 *b 3 X 3 +b 4 X 4 *b 5 X 5 +b 6 X 6 *e 11 ; 
Y = b 1 (X 1 +X 2 )+b 3 (X 3 -X 2 )+b 4 (X 4 +X 2 )+b 5 X 5 +b 6 X 6 +e ir (14) 

Th«r using , D, and D 3 as previously defined, Y = b i D i +b 3 D 2 +b 4 D 3 +b 5 X 5 +b 6 X 6 +e l 1" ^ 15 ^ 



R? c = .66038 and 
lb 

2 2 

F = (R 10" R 15 )/1 ' .7U83-.66038 =. .05145 = 4.29. 

. ,,. R 2 o)/(N . 6) (l-.71183)/24 (l-.71183)/24 

t = «/F.= 2.07. Since, two contrasts are planned, a value of 2.39 is necessary 
for significance of the .05 level, hence the hypothesis ^"''l 4~^3' corres " 
ponding to Y 2 "^1 = V^3 cannot be re J ected - The imposition of the second 
restriction (b 5 =bg) yields: 

Y = b 1 X 1+ b 2 X 2+ b 3 X 3+ b 4 X 4+ bgX 5+ b 6 X 6+ e l2 ; 

Y = b 1 X 1+ b 2 X 2+ b 3 X 3+ b 4 X 4+ b 5 (X 5+ X 6 ) + e 12> (16) 
Using D^, 

Y = J» 1 X 1 *b 2 X 2+ b 3 X3 + b 4 X 4+ b 5 D 4 *6 lz . 

R 2 1? ■ .48028 and F = (rJ q -rJ 7 )/1 = .71183-. 48028 = 23155 . 19 28) 



\ 



(i-RfoWTeT ('-»"»)/» U-««»J/** 

t = /F = 4.39, t>3.09 from Dunn's table, so that p<.01, Note also that from 
She numerator of these two tests that .05145 + .23155 = .28300, within rounding 

error of the numerat or when both restrictions were applied; this is Decause 

these contrasts are independent. From these calculations, it can be seen that 
the greatest portion of the rejection of the hypotheses tested ^by the restric- 
tions in equation 13 is due to the differences in the groups that were posttested 
only rather than due to differential increases. 

A second set of restrictions (actually, a single restriction) is given as 

(k -b )-(b,-bJ s b c -b*. This restriction tests the hypothesis related to 
2 1 4 3 b r 

(7 -Y )-(Y,-YJ*Y,-\ > that is, tne difference between £he mean of the gain 
w 2 1 M "3 y 6 o 

scores is equal to the difference in posttest measures of the non-pretested 
qroup. The restriction can be stated as b 2 " V b 6 +b l + V b 3' Im R osin 9- this 
restriction yields: 

. \ - b l X l + i b 5" b 6 +b l +b 4- b 3 )X 2 +b 3 X 3 hb 4 X 4- +b 5 X 5 +b 6 X 6 +e 13 ; (18) 
ERXC Y ' s b 1 (X 1+ X 2 ) + b 3 (X 3 -X 2 )+b 4 (X 4+ X 2 ) + b 5 (X 5+ X 2 ) + b 6 ()( 6 -X 2 ) + e 13 (19) 



Using 0., D 2 , D 3 and defining D g « Xg+Xg-and D g = X g -X 2 , equation 19 can be 
rewritten as Y = ^D^D^b^+b^+bgDg+e^ ^ 
r| q =. 70326. 

Then F = (.71183-. 70326)/! = .01857 = 1.55, N *■ 

(l-.71183)/24 (l-.71183)/24 

which is non-significant. Thus-, w.hile we have previously showed that the 
differences between the posttested groups it significant (p<.01) and the 
differences in qains in the pretested. groups are non-significant (p>.05), 
ther? are no significant differences between the gain of the mean scores 
and the posttested only groups differences. This is not to say the outcomes 
for the S lomon design are uninterpretable ; it does say that the inter- 
pretations are tricky. 

Viewing the Solomon as a Three-Way Design 
It is possible to view the Solomon design* as a 2x2x2 design with two 
missing cells. The missing cells are planned , as was the case in m4ssing 
cell design described by Williams and Wali (1979). In diagramatic .form, 



the three dimensional case can be seen asr 

Pretested 



Pre p ost 



Non-Pretested 
Pre Post 



Experimental 
Control 



Group 
i 
i 



Group 
2 



Group 

' 3 



Group 

4 fl 



X 


Group 
5 


1 X 


Grouo 
6 



r o test for the experimental -control main effect (A effect), the following 



rpstriction can be imposed; 

1 

b l +b 2 +b 5 =b 3 + V b 6 ' 



which yields 

Y - b^-XjJ+b^^ (2D 
Defining D ? = X 2~*l ; 



11 



ERIC 



3 



D 9 = X 4 +X f'- 
D l0 = X 5 -X lS and 

D H " V X 1 



-Y = b 2 D 7+ b 3 D 8+ b 4 D 9+ b 5 D 1()+ b 6 D 11+ e l4 . (22) 
R 2 22 = .29159; . . 

F « (.71183-.29159)/1 = .42024 = 35.00, p<.01. 
(l-.71183)/24 .28817/24 

To test the effect of pretesting (.the B effect), several rival' hypotheses 

\ 

might be used to serve as the main effect. 

fine such hypothesis is b 1 + b 2 +b 3 +b 4 csb 5 +b 6 i This h yP° the ^ is does not 
test the more appropriate hypothesis of interest, since the pretested scores 
are being compared to the scores wnich have been posttested only.. More inter- 
esting is b^+b 4 =b 5 +b 6 or b 2 =b 5 +b & -b 4 . 
Then, 

Y = b 1 X 1+ b 3 X 3+ b 4 (X 4 -X 2 ) + b 5 (X 5+ X 2 ) + b 6 (X 6+ X 2 ) + e l5 . (23) 
Defining D^ 2 = ^-X 2 ; 

D 13 = ^ 5 +X 2 ; . ' 



D 14 - X 6+ X 2 , 



Y = 



b 1 X 1+ i> 3 X 3+ b 4 D l2+ b 5 D 1 3 + r 6 D 14T e l5 . (24) 



R 2 2A = .69897; • 

F = (. 71183-. 69897)/! = .01286 > = -1.07, p>.05. 

(l-.71183)/24 • (l-.71183)/24 

The outcome of this test would suggest 'that the effect of pretesting per se fs 
ir.inimal for this data set. 

To test for pre-post differences (the C main effect), the restriction 

b,+b 3 =b 2 +b 4 or bj ^ 2 +b 4" b 3 ca ^3^ imposed. Then 

Y - (b 2 +b 4 -b 3 )X 1+ b 2 X 2+ b 3 X 3+ b 4 X 4+ b 5 X 5+ b 6 X 6+ e 16 , or . - 

Y - b 2 (X 2+ X,) + b 3 (X 3 -X 1 ) + b 4 (X 4+ X 1 ) + b 5 X 5+ bgX 6+ e 16 (25) 



12 




Letting D 15 » Xj-Xj, e 9*** ion 25 can be rewritten 
Y = b 2 D 1+ b 3 D 15+ b 4 D 9+ b 5 X 5+ b 6 X 6+ e 16 . (26) 
R 2 g =.506G0; 



. 71183-, 50600)/! ■ . 20583 = 17.14, p<.01, 
71183)/24 Tl-. 711831724 * 



indicating a pre-test increase in scores. 

\ 

In teractions in the Three -Way Design ^ 
First of all, the two missing cells will cause the non-existence of 
two interactions. The three way interaction will not exist, since it is 
impossible to have non-pre£&ste^groups who were pretested. For the same 
reason, the BC interaction will fail to exist. To test for the AB inter^ 
s action, that is, the interaction between the experimental -control condition 

(A) and the effect of pretesting (B), the, restriction on the full model 'Would be: 
b 2 -b 5 ■ b 4 -b 6 or b 2 = b 4 -b 6 +b 5 . 

Then Y = b 1 X 1 +(b 4 -b 6+ b 5 )X 2+ b 3 X 3+ b 4 X£b 5 X 5+ b 6 X 6+ e 17 , or 
Y = b 1 X^h 3 X 3+ b 4 (X 4+ X 2 ) + b 5 fX 5+ X 2 .)+t((X 6 -X 2 ) + e l7 {2?) 

Using previously defined transformations, Y = b^+b^+b^+bgD^+bgDg+e^ (g8) 
R 2 7 ■- .71183; R? 3 is identical to th-. R 2 for the full model. This is circum- 
stantially so bciftuse VYVY 13- 12=7-6., Thus, the AB interaction is 
< equal to zero. 

• To test the AC interaction, that is, the experimental -control condition (A) 
with pre-posfc difference's (C), the restriction bg-bj-b^bj would be imposed on 
the full model. This in fact was already done in equation 15, yielding R 15 =. 66038, 
F - 4.29, p>.05. The results from- the three-way analysis can be placed into 
a summary table"; see" Table 4: 
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Table 4 

Summary Table for a Three-Way Solution to the Solomon Design 



' ; Effect 
F .11 Model 

A (experimental-control) 

\ 

C (pre-post differences) 



3 (pretesting) 



t 



Restriction 

' b l +b 2 +b 5 =b 3 +b 4 +b 6 
b 2 +b 4 =b 5 +b 6 
b l +t 3 =b 2 +b 4 
b 2" b 5 =b 4- b 6 
b 2 _b l =b 4" b 3 



.29159 
.69897 
. 50600 
.71183 
.66038 



df 



SS 
163.33 
5.00 
80.00 
0 

20.00 



Deviation from Full Model 



.28817 24 112.00 



MS 
163.33 
5.00 
80.00 
0 

20.00 
4.67 



Finding the sum of squares in Table 4 is facilitated by knowing SS T = 388.67. 
Also, the C effect and the AC effect are identical to the same effects as 
shown in Table 3. 



F 

35.00 
1.07 
17.14 
0 

4.29 



r 
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